Configurable Three-Dimensional Neural Network Array

ABSTRACT

Configurable three-dimensional neural network array. In an exemplary embodiment, a three-dimensional (3D) neural network array includes a plurality of stacked synapse layers having a first orientation, and a plurality of synapse lines having a second orientation and passing through the synapse layers. The neural network array also includes synapse elements connected between the synapse layers and synapse lines. Each synapse element includes a programmable resistive element. The neural network array also includes a plurality of output neurons, and a plurality of select transistors connected between the synapse lines and the output neurons. The gate terminals of the select transistors receive input signals.

CLAIM TO PRIORITY

This Application is a continuation of co-pending U.S. patent applicationSer. No. 16/006,727, filed on Jun. 12, 2018, and entitled “CONFIGURABLETHREE-DIMENSIONAL NEURAL NETWORK ARRAY.” The application Ser. No.16/006,727 claims the benefit of priority based upon U.S. ProvisionalPatent Application having application Ser. No. 62/619,800, filed on Jan.20, 2018, and entitled “3D NEURAL NETWORK ARRAY” and U.S. ProvisionalPatent Application having application Ser. No. 62/622,425, filed on Jan.26, 2018, and entitled “HIGHLY CONFIGURABLE 3D NEURAL NETWORK ARRAY” andU.S. Provisional Patent Application having application Ser. No.62/570,518, filed on Oct. 10, 2017, and entitled “3D Neural NetworkArray” and U.S. Provisional Patent Application having Application No.62/572,411, filed on Oct. 13, 2017, and entitled “3D Neural NetworkArray” and U.S. Provisional Patent Application having application Ser.No. 62/574,895, filed on Oct. 20, 2017, and entitled “3D Neural NetworkArray” and U.S. Provisional Patent Application having application Ser.No. 62/577,171, filed on Oct. 26, 2017, and entitled “Neural NetworkTraining Algorithm” and U.S. Provisional Patent Application havingapplication Ser. No. 62/617,173, filed on Jan. 12, 2018, and entitled“3D Neural Network Array” all of which are hereby incorporated herein byreference in their entireties.

FIELD OF THE INVENTION

The exemplary embodiments of the present invention relate generally tothe field of semiconductors, and more specifically to the design andoperation of semiconductors forming neural network arrays.

BACKGROUND OF THE INVENTION

A neural network is an artificial intelligence (AI) system that haslearning capabilities. AI systems have been used for may applicationssuch as voice recognition, pattern recognition, and hand-writingrecognition to name a few.

The typical neural network having neurons connected by synapses may beimplemented by using software or hardware. A software implementation ofa neutral network relies on a high-performance CPU to execute specificalgorithms. For very high density neural networks, the speed of the CPUmay become a bottleneck to the performance of real-time tasks. On theother hand, a hardware implementation typically results in circuit sizesthat may limit the density or size of the neural network therebylimiting its functionality.

Therefore, it is desirable to have a configurable 3D neural networkarray that overcomes the problems of conventional arrays.

SUMMARY

A configurable three-dimensional neural network array is disclosed. Invarious exemplary embodiments, a 3D neural network array includes aplurality of network layers that are interconnected and can beconfigured through pass gates. By configuring the array, any number ofneurons can be configured to perform a particular task. This also allowsone array structure to be easily sub-divided into multiple arrays. Inaccordance with the exemplary embodiments, the output neurons of onenetwork layer are directly connected to select gates of an adjacentlayer without the use of operational amplifiers or comparator. Theresult is a fast, high density configurable neural network array.

In an exemplary embodiment, a three-dimensional (3D) neural networkarray is disclosed that includes a plurality of stacked synapse layershaving a first orientation, and a plurality of synapse lines having asecond orientation and passing through the synapse layers. The neuralnetwork array also includes synapse elements connected between thesynapse layers and synapse lines. Each synapse element includes aprogrammable resistive element. The neural network array also includes aplurality of output neurons, and a plurality of select transistorsconnected between the synapse lines and the output neurons. Gateterminals of the select transistors receive input signals.

In an exemplary embodiment, a method for generating a 3D neural networkarray is disclosed. The method comprises operations of forming outputneuron layers, and forming select gates on the output neuron layers. Theselect gates cross from a first output neuron layer to an adjacentoutput neuron layer. The method also comprises forming vertical channelson the select gates and forming landing pads on top of the verticalchannels. The method also comprises forming multiple synapse layers ontop of the select gates, and forming synapse lines through the synapselayers. The synapse lines connect to corresponding landing pads.

Additional features and benefits of the exemplary embodiments of thepresent invention will become apparent from the detailed description,figures and claims set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

The exemplary embodiments of the present invention will be understoodmore fully from the detailed description given below and from theaccompanying drawings of various embodiments of the invention, which,however, should not be taken to limit the invention to the specificembodiments, but are for explanation and understanding only.

FIG. 1A shows an exemplary embodiment of a neural network structure;

FIG. 1B shows an exemplary embodiment that illustrates basic functionsprovided by a neuron;

FIG. 2A shows an exemplary embodiment of a 3D neural network array;

FIGS. 2B-C show two exemplary embodiments of synapse structures;

FIGS. 2D-E show exemplary embodiments of 3D neural network arrays;

FIG. 3 shows an exemplary embodiment of two connected neuron layers of aneural network;

FIG. 4A shows an exemplary embodiment of an equivalent circuit of thearray shown in FIG. 3;

FIG. 4B shows an exemplary embodiment of an equivalent circuit of asynapse element;

FIG. 4C shows an exemplary current-to-voltage (I-V) curve illustratingthe operation of a resistive element that is part of the synapse elementshown in FIG. 4B;

FIGS. 5A-B show exemplary graphs that illustrate the threshold behaviorof NMOS and PMOS select transistors, respectively;

FIG. 6 shows another exemplary embodiment of a neural network array inwhich pull-up devices are added to the output neurons;

FIGS. 7A-B show exemplary embodiments of graphs that illustrate thethreshold behavior of the select transistors using NMOS and PMOS,respectively

FIGS. 8A-D show an exemplary embodiment of process steps forconstructing a 3D neural network array;

FIG. 9 shows an exemplary embodiment of a top view of the selecttransistors and output neurons as shown in FIG. 8B;

FIG. 10 shows an exemplary embodiment of a 3D neural network array wherethe select transistors are located on the top of the array;

FIG. 11 shows an exemplary embodiment of two neuron layers of a neuralnetwork;

FIG. 12 shows a detailed exemplary embodiment of the array shown in FIG.11;

FIG. 13 shows another exemplary embodiment of the array shown in FIG.11;

FIG. 14A shows an exemplary embodiment of an equivalent circuit of the3D neural network array shown in FIG. 12;

FIG. 14B shows an exemplary embodiment of a synapse;

FIG. 15A shows an exemplary embodiment of a bottom structure of the 3Dneural network array shown on FIG. 12;

FIG. 15B shows an exemplary embodiment of the bottom structure of the 3Dneural network array shown in FIG. 13;

FIG. 16A shows an exemplary embodiment of a top view of a bottomstructure similar to that shown in FIG. 15B;

FIG. 16B shows an exemplary embodiment of the top view shown in FIG. 16Aand illustrates the connection of the pass gates;

FIG. 17A shows an embodiment of using the pass gates to configure theneural network illustrated in FIG. 16A;

FIG. 17B shows another embodiment of configuring the neural networkillustrated in FIG. 16A;

FIG. 17C shows another embodiment of configuring the neural networkillustrated in FIG. 16A;

FIG. 17D shows another embodiment of configuring the neural networkillustrated in FIG. 16A; and

FIG. 18 shows an exemplary embodiment of a programming system suitableto directly program 3D neural networks arrays.

DETAILED DESCRIPTION

Those of ordinary skilled in the art will realize that the followingdetailed description is illustrative only and is not intended to be inany way limiting. Other embodiments of the present invention willreadily suggest themselves to skilled persons having the benefit of thisdisclosure. Reference will now be made in detail to implementations ofthe exemplary embodiments of the present invention as illustrated in theaccompanying drawings. The same reference indicators or numbers will beused throughout the drawings and the following detailed description torefer to the same or like parts.

FIG. 1A shows an exemplary embodiment of a neural network structure. Theneural network may contain multiple neuron layers, such as layers 110a-c. Each neuron layer may contain input neurons and output neurons. Forexample, layer 110 a has input neurons 101 a-c, and output neurons 102a-d. Each input and output neuron of a layer are connected by synapsessuch as synapses 103 a-c. The synapses comprise a certain ‘weight’ thatis applied to signals flowing through the synapse.

FIG. 1B shows an exemplary embodiment that illustrates basic functionsprovided by a neuron. For example, the neuron provides a SUM function104 a that sums signals from the previous layer's neurons and theweights, and a ACTIVATION function 104 b that generates a non-linearoutput.

FIG. 2A shows an exemplary embodiment of a 3D neural network array. Thearray shown in FIG. 2A comprises multiple conductor layers (e.g., 201a-h) called SYNAPSE LAYERS, and multiple vertical conductor layers(e.g., 202 a-d), called SYNAPSE LINES. At the intersection of synapselayers and synapse lines are SYNAPSES (e.g., synapse 200). The synapsestructure is shown in FIGS. 2B-C. The array also comprises selecttransistors (e.g., 203 a-d), which may be NMOS or PMOS, junction-less orjunction devices. The gates of the select transistors are connected toinput signals (INPUTS), which are provided by the outputs from aprevious layer. Also shown for this layer are OUTPUT NEURONS (e.g., 204a-d). During operation, the select transistors may be turned on to allowcurrent to flow from the synapse lines to the output neurons.

FIGS. 2B-C show two exemplary embodiments of synapse structures. Forexample, the synapse structures shown in FIGS. 2B-C are suitable for useas synapses in the array shown in FIG. 2A, such as the synapse 200. Thesynapse structures comprise synapse layer 220 and synapse line 221,which comprise a specific metal, such as Pt, Ti, Ta, Cu, or anothersuitable conductor. The synapse structures also comprise a resistiveelement 222, which comprises a material such as HfOx, TaOx, PtOx, TiOx,or phase-change element, ferroelectric material, or magnetic material.The resistance of the resistive element 222 (or phase-change element)may be changed by applying the proper bias conditions on the synapselayer 220 and synapse line 221. The synapse structures also comprise aselector 223, which comprises a diode, Schottky diode, or otherthreshold-behavior materials such as a diode material, Schottky diodematerial, NbOx material, TaOx material and VCrOx material. The diode maybe unidirectional or bi-directional. It should be noted that for a 3Dneural network array, a selector device for each synapse may be used toprevent sneak current leakage path through the unselected synapse layersand synapse lines.

FIGS. 2D-E show exemplary embodiments of 3D neural network arrays.

In FIG. 2D the synapse layers, such as layer 201 a may be separated intomultiple horizontal lines, such as horizontal lines 210 a to 210 d. Theeven lines 210 a and 210 c may be connected together (not shown), andthe odd lines 210 b and 201 d may be connected together (not shown).When the synapse layer 201 a is selected, the even lines 210 a and 210 cmay be supplied with VH to represent ‘positive weights’ and the oddlines 210 b and 210 d may be supplied with VL to represent ‘negativeweights’.

In FIG. 2E the synapse layer such as layer 201 a may be separated intomultiple horizontal lines, such as lines 210 a to 210 e. The verticalsynapse lines, such as lines 202 a to 202 d, are formed between thehorizontal lines instead of being trenched through the horizontal lines.For example, the vertical synapse line 202 d is formed between thehorizontal lines 210 a and 210 b. This structure forms two synapses, asshown at 214 a and 214 b in the junctions between vertical line 202 dand horizontal lines 210 a and 210 b. The even lines 210 a, 210 c, and210 e may be connected together (not shown), and the odd lines 210 b and201 d may be connected together (not shown). When the synapse layer 201a is selected, the even portions 210 a, 210 c, and 210 e may be suppliedwith VH to represent ‘positive weights’ and the odd portions 210 b and210 d may be supplied with VL to represent ‘negative weights’.

FIG. 3 shows an exemplary embodiment of two connected neuron layers of aneural network. As illustrated in FIG. 3, neuron layer 301 a isconnected to neuron layer 301 b. For example, the output neurons 204 a-dof the first layer 301 a are connected to the input neurons 205 a-d ofthe layer 301 b through contacts (e.g., contacts 206 a-b). In accordancewith exemplary embodiments, the output neurons 204 a-d are directlyconnected to the select gates of the next layer's inputs without the useof operational amplifiers or comparators. This significantly reducescircuit size and results in a very compact neural network array. Thisalso allows the synapses layers 201 a-h and 201 a′-h′ to be connected.For ease of understanding, FIG. 3 shows the synapses layers 201 a-h and201 a′-h′ separated so that is easier to view the connection of theoutput neurons 204 a-d to the select gates 205 a-b, however, in anactual implementation, the synapses layers 201 a-h and 201 a′-h′ areconnected.

FIG. 4A shows an exemplary embodiment of an equivalent circuit of thearray shown in FIG. 3. For example, the circuit shown in FIG. 4Aincludes inputs 203 and outputs 204 for the first network layer 301 a,and inputs 205 and outputs 207 for the second network layer 301 b. Alsoshown are contacts 206 between the outputs 204 and the inputs 205.

FIG. 4B shows an exemplary embodiment of an equivalent circuit of thesynapse element. For example, the synapse shown in FIG. 4B is suitablefor use as the synapse 200 shown in FIG. 4A. The synapse comprises aselector 401, which may have bidirectional or unidirectional thresholdbehavior, and a resistive element 402.

FIG. 4C shows an exemplary current-to-voltage (I-V) curve illustratingthe operation of a resistive element (e.g., resistive element 402). Asillustrated by the I-V curve, Vt and Vt− are the threshold voltage ofthe selector 401, and VSET and VRES are the voltage levels to programthe resistive element 401 to low and high resistances, respectively. Forease of understanding, it will be assumed that VT=0.5V, VT−=−0.5V,VSET=2.5V, and VRES=−2.5V.

Referring again to FIG. 4A, a detailed description of the operation ofthe 3D neural network is provided. The multiple synapse layers (e.g.,201) may store the synapse values (e.g., resistive element values) fordifferent tasks or functions, such as, for example, image recognition orvoice recognition. During operation, only one task is selected at atime. For example, each task can be stored in two synapse layers. Theselected two layers are supplied with VH and VL, respectively, as shownin FIG. 4C to represent ‘positive weights’ and ‘negative weights’ of thesynapses. All the other unselected layers are supplied with VDES.

Referring now to FIG. 4C for the levels of VH, VL, and VDES. VH must behigher than VT to conduct current from the synapse layer to the synapselines. VL must be lower than VT to allow current to flow from thesynapse lines to the synapse layer. Due to the threshold function, thevoltage of the synapse lines (e.g., 202) will be between (VL+Vt) and(VH−Vt). For example, assuming VH and VL are 2V and 0V, respectively,the synapse line's voltage will be between 0.5V to 1.5V. The level ofVDES may be 1.0V to prevent leakage current.

Referring again to FIG. 4A, the current of synapse lines 202a-m willpass through the select transistors 203 a-m and ‘sum’ together in theoutput neuron 204 d. The currents of the select transistors 203 a-m aredetermined by the input voltages IN[0] to IN[m]. When the input voltageis below the threshold voltage of the transistor, the transistor willturn off and thus no current will flow through.

When voltage is passed from the synapse lines to output neurons, the Vtdrop of the select transistor may cause the output level to be lowerthan the synapse lines. When the output neurons' voltages are applied tothe gates of the select transistors of the next layer, another Vt dropoccurs. This causes the output voltage to degrade.

In an exemplary embodiment, the select transistors are formed using‘native’ devices. The Vt of the native device is close to OV. Therefore,when the synapse lines' voltage, such as 0.5V to 1.5V is passed to theoutput neurons, the full voltage level of 0.5V to 1.5V can be passed.Therefore, when the voltage is applied to the next layer's selecttransistors (e.g., 205), it can fully pass the next layer's synapselines' voltage to the output neurons. According to another exemplaryembodiment, the select transistors (e.g., 203) may be low Vt devices ordepletion devices.

It should be noted that the ‘activation function’ of the neurons isperformed by the select transistors, which results in a very compactarray size. In comparison, conventional arrays usually use operationamplifiers or comparators to perform the threshold function of theneurons, thus require very larger circuit size and limit the number ofthe neurons of a chip.

FIGS. 5A-B show exemplary graphs that illustrate the threshold behaviorof NMOS and PMOS select transistors, respectively. It should be notedthat in these exemplary graphs, the INPUT, SYNAPSE lines, and OUTPUTvoltages are all in the same range from 0.5V to 1.5V.

FIG. 6 shows another exemplary embodiment of a neural network array inwhich pull-up devices 211 a-cand 212 a-d are added to the output neurons204 a-d and 207a-d, respectively. In this embodiment, the selecttransistors comprise enhancement devices. For example, when thetransistor charges the output neuron to (IN−Vt), the transistor isturned off. The pull-up device then charges the output neuron to thefull voltage range such as 2V, thus this configuration compensates forthe Vt drop. The pull-up device may be located on the substrate underthe 3D array, or formed by using the same type of transistor as thevertical select transistors shown. The pull-up transistors may be PMOSor NMOS. For PMOS, the pull-up device's current will be weaker than thepull-down current of the synapses. For a PMOS embodiment, the gate maybe connected to a bias voltage to limit the pull-up current. For an NMOSembodiment, a pre-charge pulse higher than 2.5V may be applied to thegate to pre-charge the output neuron to 2V.

In another exemplary embodiment using PMOS as the select transistors,the devices 211 a-cand 212 a-d are pull-down devices to fully dischargethe output neuron to OV when the select transistors are turned off.

FIGS. 7A-B show exemplary embodiments of graphs that illustrate thethreshold behavior of the select transistors using NMOS and PMOS,respectively. It should be noted that the output voltage may be pullhigh to 2V for FIG. 7A and pull low to OV for FIG. 7B.

FIGS. 8A-D show an exemplary embodiment of process steps forconstructing a 3D neural network array.

FIG. 8A shows an exemplary embodiment of the output neuron layers of the3D neural network array, which may be formed by a conductor layer suchas diffusion, polysilicon, or metal. The output neuron layers comprise afirst layer 801 a-d, a second layer 802 a-d, a third layer 803 a-d, afourth layer 804 a-d, and a fifth layer 805 a-d. Also shown are contacts807 a-d that are used to connect the output neurons to the gates of theselect transistors of the next layer.

FIG. 8B shows an exemplary embodiment illustrating how select gates 811a-d, 812 a-d, 813 a-d, 814 a-d, 815 a-d, and 816 a-d are formed on theoutput neuron layers. Note that the select gates cross one output neuronlayer and connect to contacts on an adjacent output neuron layer. Forexample, the select gate 816 d crosses the output layers 806 a-d toconnect with contact 807 e. Next, vertical channels, such as 817 a-d,are formed on the select gates.

FIG. 8C shows an exemplary embodiment illustrating how landing pads,such as landing pads 820 a-d are formed on top of the vertical channels.

FIG. 8D shows an exemplary embodiment illustrating how multiple synapselayers, such as synapse layer 821 a-h are deposited on top of the selecttransistors. Multiple vertical synapse lines, such as 822 a-d also areformed through the synapse layers to connect to the landing pads.

FIG. 9 shows an exemplary embodiment of a top view of the selecttransistors and output neurons as shown in FIG. 8B. This top view showsthe first layer 901 a to the sixth layer 901 f. Also shown is an outputneuron layer 902 that comprises a conductor such as diffusion,polysilicon, or metal. Select gate 903 is shown and contact 904 connectsthe output neuron 902 and the select gate 903. Also shown are verticalsynapse lines, such as vertical synapse line 905.

Please notice, although the exemplary embodiments show arrays usingvertical select transistors, it is obvious that the embodiments can beimplemented by using any type of select transistors, such as planartransistors or FinFET. Such variations are within the scope of theexemplary embodiments.

FIG. 10 shows an exemplary embodiment of a 3D neural network array wherethe select transistors 203 and 205 are located on the top of the array.In another exemplary embodiment, decoders may be added to the disclosedarray architecture to select each neuron in order to apply biasconditions to set and reset the synapses.

FIG. 11 shows an exemplary embodiment of two neuron layers of a neuralnetwork. The neural network comprises neuron layers 1100 a and 1100 b. Atop portion 1110 a comprises multiple synapse layers 1101 a-h. A bottomportion 1110 b comprises input neurons 1103 a-d and 1105 a-d, and outputneurons 1104 a-d and 1107 a-d. The output neurons 1104 a-d of the firstlayer 1100 a are connected to the input neurons 1105 a-d of the secondlayer 1100 b through contacts, such as contacts 1106 a-b. In variousexemplary embodiments, the output neurons of one network layer aredirectly connected to the select gates of the next network layer's inputneurons without using operational amplifiers or comparators. Thissignificantly reduce the circuit size and results in very compact neuralnetwork arrays. In this application, the output neurons of a networklayer are solid-connected to the input neurons of the next networklayers. Therefore, the number of the input and out neurons in each layeris fixed.

Alternatively, in other exemplary embodiments, pass gates are addedbetween the output neurons and the next network layer's input neurons.This provides increased flexibility in configuring the neural network'snumber of layers and neurons in each layer. It also provides anaccessibility that each layer's neurons may be selectively controlled byperiphery circuits for program operation.

FIG. 12 shows a detailed exemplary embodiment of the array shown in FIG.11. In FIG. 12, the output neurons 1104 a-d of the first network layer1100 a are connected to the input neurons 1105 a-d of the next networklayer 1100 b through vertical pass gates 1201 a. When the vertical passgates 1201 a are turned on, the output neurons' voltages are passed tothe conductor layers 1202 a-d that are connected to the input neurons1105 a-d through contacts.

Similarly, the output neurons 1104 a-d may be connected to input neuronsof an adjacent network layer (not shown) through vertical pass gates1201 b and the conductor layers 1203 a-d.

Similarly, the input neurons 1105 a-d may be connected to output neuronsof an adjacent network layer (not shown) through the conductor layer1204 a-d. The conductor layer 1202 a-d may be the same metal layer aslanding pads (not shown) on each vertical channel of the select gates,such as vertical channel 1205. The output neurons 1104 a-d and 1107 a-dmay be formed by conductor layers, such as metal or polysilicon, ordiffusion layer. At least one advantage of this embodiment is that whenusing metal layers as the output neurons, the entire 3D array includingthe bottom structure may be located on top of other circuits, such asCPU, for example. Thus, the footprint for the 3D neural network arraywould require no additional silicon.

FIG. 13 shows another exemplary embodiment of the array shown in FIG.11. In this embodiment, the output neuron layers 1104 a-d and 1107 a-dare diffusion layers on the substrate. The pass gates 1201 a-b areplanar transistors.

FIG. 14A shows an exemplary embodiment of an equivalent circuit of the3D neural network array shown in FIG. 12. Please notice, the outputneurons 1104 a-d are connected to the input neurons 1105 a-d of the nextnetwork layer through the pass gates (1201 a). A synapse 1400 is alsoshown.

FIG. 14B shows an exemplary embodiment of a synapse. For example, thesynapse shown in FIG. 14B is suitable for use as the synapse 1400 shownin FIG. 14A. The synapse comprises a selector 1401 and a resistiveelement 1402 that alternatively can be a phase-change element.

FIG. 15A shows an exemplary embodiment of a bottom structure of the 3Dneural network array shown on FIG. 12. The bottom structure comprisesmultiple neuron layers, such as neuron layers 1500 a-c. Each layer maybe connected to adjacent layers through pass gates, such as pass gates1501 a-b. In this embodiment, the pass gates are formed by verticaltransistors. Also shown are input neurons 1502 a-d, output neurons 1503a-d, and conductor layers 1504 a-d.

FIG. 15B shows an exemplary embodiment of the bottom structure of the 3Dneural network array shown in FIG. 13. In this bottom structure, thepass gates 1505 a-b are formed by planar transistors. The output neurons1503 a-d are formed by diffusion layers. The output neurons may beconnected to the adjacent network layers' input neurons through contacts1506 a-d.

FIG. 16A shows an exemplary embodiment of a top view of a bottomstructure similar to that shown in FIG. 15B. However, for FIG. 16A a 4x4network layer structure is shown. It also should be noted that similarprocess operations may be applied to the embodiment shown in FIG. 15A.

FIG. 16B shows an exemplary embodiment of the top view shown in FIG. 16Aand illustrates the connection of the pass gates. For ease ofunderstanding, only the pass gates are shown. The pass gates may beconnected to vertical signal lines 1601 a to 1601 j and horizontalsignal lines 1602 a to 1602 j as shown. The signal lines may be formedby metal layers. The signals lines may be connected to decoder circuitsto selectively turn the pass gates on or off. It should be noted thatthe connections shown in FIG. 16B are exemplary and it should be notedthat the pass gates may be connected in many other suitable ways. Forexample, in another exemplary embodiment, the number of vertical signallines may be doubled, thus the select gates 1603 a to 1603 d may beconnected to different signal lines to provide higher flexibility.

FIG. 17A shows an embodiment of using the pass gates to configure theneural network illustrated in FIG. 16A. The pass gates being turned onare shown in grey color. The pass gates being turned off are shown inblack color. The vertical pass gates such as 1701 a and 1701 b set theboundary of the neural network. The horizonal pass gates such as 1702 aand 1702 b set the direction of the neural network. As a result, amultiple-layer neural network is configured as shown by the arrow line1703. For example, the arrow 1703 shows the signal flow through theneural network layers based on the enabled pass gates.

FIG. 17B shows another embodiment of configuring the neural networkillustrated in FIG. 16A. By changing the boundary 1701 a and 1701 b, theneurons of each layer may be increased or decreased. For example,compared with the neural network in FIG. 17A, the neural network shownin FIG. 17B has twice the number of neurons in each layer, asillustrated by the arrow lines 1703 a and 1703 b.

FIG. 17C shows another embodiment of configuring the neural networkillustrated in FIG. 16A. In this embodiment, the neural network pathshown by the arrow line 1703 has feedback loops 1706 a and 1706 b. Thus,this neural network forms a ‘concurrent’ neural network.

FIG. 17D shows another embodiment of configuring the neural networkillustrated in FIG. 16A. In this configuration, the input neurons andoutput neurons of the selected layer 1720 may be accessed by peripherycircuits, as shown in arrow lines 1721 a and 1721 b, respectively. Theinput lines of 1722 a to 1722 d may be supplied with OV to turn off theselect gates along the path of arrow lines 1721 a and 1721 b. The passgates shown in black such as 1723 a and 1723 b may be turned off toisolate the selected area. Thus, other parts of the array may be usedfor other operations during this operation. This mode may be used toprogram the synapses of the selected layer.

In an exemplary embodiment, it is possible to ‘directly training theneural network’ without using a traditional back-propagation algorithm.The approach directly selects a neuron and changes its output higher orlower to see which direction reduces the output error. Then, thesynapses connected to this neuron are changed accordingly. These stepsare repeated for all the neurons until all the synapses are changed.This is called an iteration. The system may repeat the iterations toreduce the output error continuously until the desired error is reached.

In another exemplary embodiment, the synapses connected to a selectedneuron are directly programmed to make the neuron's output higher orlower. A check is performed to compare the output to a target value todetermine an output error. If the output error is reduced, the systemmay select the next neuron and repeat the operations. If the error isincreased, the system may apply a reverse-program to the synapsesconnected to the neuron, to reduce neuron's output, and then check theoutput error. If the error is reduced, the system may select the nextneuron and repeat the operations. By using this approach, themathematically heavy computations used for back-propagation can beeliminated.

In various exemplary embodiments, the 3D neural network arrays disclosedherein are suitable for programming by a direct training approach.

FIG. 18 shows an exemplary embodiment of a programming system suitableto directly program 3D neural networks arrays. As illustrated in FIG.18, a neural network array 1801 is programmed by the programming system.The inputs 1802 are fed into the neural network to generate the outputs1803. The outputs are compared with target values 1804 by an outputerror comparator 1805. The neuron/layer decoder 1806 selects a layer anda neuron to allow a program control circuit 1807 to access the neuron.The program control circuit may change the selected neuron's output, andcheck the output error comparator to determine the direction of thechange. If the output is within a selected range, the program controlcircuit 1807 programs the synapses connected the neuron by appropriatelybiasing them. After that, the neuron/layer decoder may select the nextneuron for programming.

While exemplary embodiments of the present invention have been shown anddescribed, it will be obvious to those with ordinary skills in the artthat based upon the teachings herein, changes and modifications may bemade without departing from the exemplary embodiments and their broaderaspects. Therefore, the appended claims are intended to encompass withintheir scope all such changes and modifications as are within the truespirit and scope of the exemplary embodiments of the present invention.

What is claimed is:
 1. A three-dimensional (3D) neural network array,comprising: a plurality of stacked synapse layers having a firstorientation; a plurality of synapse lines having a second orientationand passing through the synapse layers; synapse elements connectedbetween the synapse layers and synapse lines, and wherein each synapseelement includes a programmable resistive element; a plurality of outputneurons; and a plurality of select transistors connected between thesynapse lines and the output neurons, and wherein gates of the selecttransistors receive input signals.
 2. The 3D neural network of claim 1,wherein the select transistors are configured as one of PMOS or NMOStransistors.
 3. The 3D neural network of claim 1, wherein the selecttransistors are configured as native transistors having little orsubstantially no voltage drop.
 4. The 3D neural network of claim 1,wherein the first orientation is horizontal and the second orientationis vertical.
 5. The 3D neural network of claim 1, wherein eachprogrammable resistive element comprises material selected from a set ofmaterials comprising resistive material, phase change material,ferroelectric material, and magnetic material.
 6. The 3D neural networkof claim 1, wherein each synapse element includes a diode.
 7. The 3Dneural network of claim 6, wherein the diodes comprise at least one ofdiode material, Schottky diode material, NbOx material, TaOx materialand VCrOx material.
 8. The 3D neural network of claim 1, furthercomprising pull-up transistors coupled to the output neurons.
 9. The 3Dneural network of claim 1, further comprising pull-down transistorscoupled to the output neurons.
 10. The 3D neural network of claim 1,wherein the output neurons are formed as surface diffusions.
 11. The 3Dneural network of claim 1, further comprising conductive contactsconnected to the output neurons.
 12. The 3D neural network of claim 10,wherein the conductive contacts further comprise in-line passtransistors.
 13. The 3D neural network of claim 12, wherein the passtransistors are formed as planar transistors.
 14. The 3D neural networkof claim 13, wherein the 3D neural network forms a first neural networklayer and wherein the conductive contacts are connected between theoutput neurons and gate terminals of select transistors associated witha second neural network layer.
 15. The 3D neural network of claim 14,wherein the pass transistors are enabled or disabled to control signalsflowing from the first neural network layer to the second neural networklayer.
 16. The 3D neural network of claim 1, wherein the 3D neuralnetwork array is stacked on a semiconductor device so that the 3D neuralnetwork array uses no additional semiconductor surface area.
 17. Amethod for generating a 3D neural network array, the method comprisingoperations of: forming output neuron layers; forming select gates on theoutput neuron layers, wherein the select gates cross from a first outputneuron layer to an adjacent output neuron layer; forming verticalchannels on the select gates, wherein landing pads are formed on top ofthe vertical channels; forming multiple synapse layers on top of theselect gates; and forming synapse lines through the synapse layers,wherein the synapse lines connect to corresponding landing pads.